Scala Macros: Let Our Powers Combine!

Compile-time metaprogramming has been proven immensely useful enabling programming techniques such as language virtualization, embedding of external DSLs, self-optimization, and boilerplate generation amongst many others. In the recent production release of Scala 2.10 we have introduced macros, an experimental facility which gives its users compile-time metaprogramming powers. Alongside of the mainline release of Scala Macros, we have also introduced other macro flavors, which provide their users with different interfaces and capabilities for interacting with the Scala compiler. In this paper, we show how the rich syntax and static types of Scala synergize with macros, through a number of real case studies using our macros (some of which are production systems) such as language virtualization, type providers, materialization of type class instances, type-level programming, and embedding of external DSLs. We explore how macros enable new and unique ways to use pre-existing language features such as implicits, dynamics, annotations, string interpolation and others, showing along the way how these synergies open up new ways of dealing with software development challenges

Unification of Compile-Time and Runtime Metaprogramming in Scala

Metaprogramming is a technique that consists in writing programs that treat other programs as data. This paradigm of software development contributes to a multitude of approaches that improve programmer productivity, including code generation, program analysis and domain-specific languages. Many programming languages and runtime systems provide support for metaprogramming. Programming platforms often distinguish the notions of compile-time and runtime metaprogramming, depending on the phase of the program lifecycle when metaprograms execute. It is common for different lifecycle phases to be hosted in different environ- ments, so it is also common for different kinds of metaprogramming to provide different capabilities to metaprogrammers. In this dissertation, we present an exploration of the idea of unifying compile-time and runtime metaprogramming in Scala. We focus on the practical aspect of the exploration; most of the described designs are available as popular software products, and some of them have become part of the standard distribution of Scala. First, guided by the motivation to consolidate disparate metaprogramming techniques available in earlier versions of Scala, we introduce scala.reflect, a unified metaprogram- ming framework that uses a language model derived from the Scala compiler to run metaprograms both at compile time and at runtime. Secondly, armed by the newfound metaprogramming powers, we describe Scala macros, a language-integrated compile-time metaprogramming facility based on scala.reflect. Thanks to the comprehensive nature of scala.reflect, macros are able to work with both syntactic and semantic information about Scala programs, enabling a wide range of previously impractical or impossible use cases. Finally, based on our experience and user feedback, we identify key strengths and weaknesses of scala.reflect and macros. We propose scala.meta, a new unified metapro- gramming framework, and inline/meta, a new macro system based on scala.meta, that take the best from their predecessors and address the most important problems

Style Checking With Scala.Meta

In many companies, code reviews are big parts of the day-to-day life of an engineer. Some can argue that the time spent on someone else’s code is time that some cannot spent on her or his own work. Code reviews usually allow to find two kinds of issues. The first kind consists of high-level design errors compared to the specifications of the program while the second type consists of style problems. In this report, we argue that the search of the later kind of error can be automated, as the set of style issues is usually well defined for a programming language. We propose modifications to a linter tool called Obey, which allows to analyze Abstract Syntax Trees of Scala code, raise warnings and suggest changes that can be automatically persisted. Over the course of this project we worked jointly with Codacy, a company proposing automated code reviews, to have an industrial application which would allow users to define their own set of rules. This report also presents the results of this real life testing

Scala Macros, a Technical Report

Metaprogramming is a powerful technique of software development, which allows to automate program generation. Applications of metaprogramming range from improving expressiveness of a programming language via deep embedding of domain-specific languages to boosting performance of produced code by providing programmer with finegrained control over compilation. In this report we introduce macros, facility that enables compile-time metaprogramming in the Scala programming language

Traversal Query Language For Scala.Meta

With the rise of metaprogramming in Scala, manipulating ASTs has become a daily job. Yet the standard API provides only low-level mechanisms to transform or to collect information on those data structures. Moreover, those mechanisms often force the programmer to manipulate state in order to retrieve information on these ASTs. In this report, we try to solve those problems by introducing TQL, a high-level combinator Scala library to transform and query data structures in a purely functional way. Parser combinators allow to combine several small parsers to build a bigger one in an expressive way. In this report, we argue that we can apply the same concept to data structure manipulation and construct complicated traversers on top of smaller ones. Yet combinators may feel unnatural or too complicated for certain usage. We therefore built a library on top of TQL to manipulate data structures as a collection. We then put TQL in practice to scala.meta ASTs, and describe the challenges we face when traversing a real-word data structure, especially performance-wise

Parser Macros for Scala

Parser macros are a new kind of macros that allow developers to create new language constructs and to define their own syntax for using them. In this report, we present why parser macros are useful and the kind of problems that they help to solve. We will also see how they are implemented and gain insight about how they take advantage from scala.meta, the new metaprogramming toolkit for Scala. Finally, we will discuss what are the current limitations of parser macros and what is left for future work

Obey: Code Health for Scala.Meta

Obey is a user-friendly tool that helps programmers enforce code health requirements in their projects. Requirements are expressed as rules, written with the TQL library combinators used to traverse scala.meta trees, that generate compiler warnings and can automatically correct the source code. Programmers are provided with a set of basic rules and can easily implement and use their own. Obey can be used to systematically enforce user-defined requirements, to automate migration between different library versions or simply to format source code. In this paper, we will present the user interface, the implementation details and the results we obtained during this semester project

Two approaches to portable macros

For any programming language that supports macros and has multiple implementations (each with different AST definitions), there is a common problem: how to make macros that operate on ASTs portable among different compiler implementations? Implementing portable macros is especially important for statically typed languages like Scala, as IDE vendors usually have different implementations of the language in order to support rich IDE features. Unportable macros compromise IDE features and degrade programming experience. We describe two approaches to the portability problem based on two different views on macros: (1) the tree-based approach, which views macros as operations on abstract syntax trees, solves the problem by defining standard abstract syntax trees; (2) the syntax-based approach, which views macros as operations on abstract syntax, solves the problem by defining standard abstract syntax. We show that the latter has significant practical advantages, especially in supporting semantic macros that use type information of ASTs to transform user code. Based on the idea, we implemented a new macro system, Gestalt, for the experimental Scala compiler, Dotty. The new implementation solves several long-standing problems of the current Scala macro system and demonstrates advantages over alternative approaches. Our solution has been adopted in the official new Scala macro system

Macros in sbt: Problem Solved!

In the previous report, we described how incremental compilation was made more complicated when macro-enabled programs are involved, and we implemented the foundations of the support for metaprograms in sbt, a build tool and incremental compiler for Scala. Since then, we worked on improving the internal representation of dependency relationships between files in sbt, to make it easier to extend sbt and to define new relationships. This crucial development allowed us to fix all the remaining problems that sbt had with macros: how should we handle their transitive dependencies? How can we know what they inspect during their expansion? Are there other means by which macros could introduce dependencies? How should we use these informations? In this report, we will expose the new techniques that have been proposed and implemented to offer a complete support for metaprograms along with all their dependencies in sbt, and explain the most relevant parts of their implementation

A TASTY Alternative

This document proposes a new serialization format for typed syntax trees of Scala programs. It aims to be ● compact: All numbers and references are length encoded, trees are inlined by default but may be shared. ● lazy: Trees can be explored from the top, and one can suspend reading subtrees at any time. The only section that needs to be scanned in full is the name table. Laziness is important for the compiler frontend as most of the information in a TASTY file is not needed then. ● extensible: New sections can be defined at will. Additional elements always carry their length, so can be safely skipped in older readers. ● precise: The format given here is in essence a serialized abstract syntax tree for typed Scala. Some of the main classifications,e.g. between types, terms and paths are reflected in the grammar. Picklers and unpicklers for the format have been implemented in https://github.com/lampepfl/dotty/pull/394. The present document gives a syntax summary of the format. More explanations are found in the TASTY reference manual